Using Visual Attention to Recognize Human Pointing Gestures in Assembly Tasks

Humans often use hand gestures to instruct other persons, e.g., to grasp an object or to look at a certain location. Especially in assembly tasks pointing gestures can simplify the cooperation between man and machine. We use visual attention to control the active cameras of our system to xate interesting image regions, e.g., assembly parts. Furthermore, the system reacts on the appearance of human hands and recognizes the direction of pointing gestures made by the user. This information is used to guide the viewing direction of the artiicial observer. We describe both, the multi-layer attention model and the hand gesture recognition. Results show the reliability of our adap-tive system, which robustly recognizes pointing gestures made by diierent users. 1 Motivation Individuals frequently use hand gestures in communication to guide the attention of another person. Especially pointing gestures often support spoken instructions or commands. Current computer input devices are mostly based on mechanical solutions like keyboards, mice or touch pads. These peripherals physically constrain the user to a very unnatural communication behavior. Nowadays, the situation improves with the advent of a new generation of voice recognition interfaces. However, there are still situations where non-verbal interaction is preferred or at least an additional modality, i.e., when referencing objects by pointing. With increasing computer power and decreasing costs for computer vision hardware (frame-grabber, cameras, etc.) used for multimedia applications, it is possible to build \seeing" interfaces. In previous work we have shown that systems using computer vision and adaptive neural network classiiers are well Figure 1: The pointing gesture of the user guides the attention of the AUC camera head to the referenced object. suited to recognize both gaze direction 7, 8] and hand gestures 6]. Limitations of such systems are still the low recognition rate in natural environments and when employed with diierent users. On the other side, even humans do not precisely recognize pointing gestures, instead they look roughly at the shown direction and than search for the nearest object, as the most likely candidate for the referenced object. In this paper, we present a user interface which recognizes pointing gestures and acquires context information about objects in the current scenario. Thus, it can use this knowledge about the object positions to determine the referenced object. Our approach consists of two modules: (i) a multi-layer model generating a cortical map (representing a saliency value at each image position) that is used …